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Introduction. Critical appraisal is a crucial aspect of evidence-based 
practice. In order to determine whether research is valid, reliable and 
applicable, the evidence-based practice process advocates that published 
research be critically appraised. Between 2006 and 2008, the journal 
Evidence Based Library and Information Practice published 101 evidence 
summaries, critically appraising research in library and information 
studies. These evidence summaries can be examined in order to determine 
common strengths and weaknesses of research relevant to library and 
information studies and identify commonalities in existing evidence 
summary commentaries. 

Method. We undertook a directed qualitative content analysis of the 
commentary portion of all 101 evidence summaries published in the 
journal, Evidence Based Library and Information Practice from 2006- 
2008. 

Findings. Evidence summaries reveal more weaknesses than strengths 
in the library and information studies research. In general, evidence 
summary writers tend to remark on weaknesses relating to validity and 
reliability, yet paradoxically point out strengths with respect to 
research's applicability to practice. 

Conclusions. Further research is required to understand why evidence 
summary writers note more weaknesses than strengths in library and 
information studies research and whether this reflects the actual quality 
of the research in general. 
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Introduction 


Critical appraisal is a crucial aspect of evidence-based practice. In order to determine whether 
research is valid, reliable and applicable, the evidence-based practice process advocates that 
published research be critically appraised. Critical appraisal in library and information studies 
is seldom published, but since 2006, the journal Evidence Based Library and Information 
Practice has published an average of eight evidence summaries in each issue. Evidence 
summaries provide a synopsis and critical appraisal of published research in order to facilitate 
the transfer of research into practice. This study aims to determine strengths and weaknesses of 
library and information studies research, as identified in the journal's evidence summaries, so 
that as a profession we might improve our research, and as readers we may consider the 
strengths and weaknesses of research publications as well as their utility. 

Background 

Evidence-based practice in library and information studies 

Evidence-based practice in library and information studies is, 

an approach to information science that promotes the collection, interpretation 
and integration of valid, important and applicable user-reported, librarian- 
observed, and research-derived evidence. The best available evidence, 
moderated by user needs and preferences, is applied to improve the quality of 
professional judgments (Booth 2000) . 

This definition stresses three main areas that contribute to evidence-based practice: 1) research 
evidence; 2) user needs and preferences; 3) professional judgments. A 11 three of these 
components are vital to evidence-based practice. Research is not conducted in isolation, but 
rather, is integrated with our professional judgment and within our local context. 

The recognized steps in evidence-based practice are: 

1. Ask a focused question 

2. Acquire evidence on the topic 

3. Appraise the research studies 

4. Apply the findings 

5. Assess the impact fDawes et al. 200^1 . 

The third step, appraise, refers to the critical appraisal of original research in order to 
determine its validity, reliability and applicability. Appraisal is a rather onerous task, requiring 
expertise in a myriad of research methods depending on the study's research design. It is this 
step that the evidence summaries published in the journal Evidence Based Library and 
Information Practice attempt to accomplish, saving the practitioners who read them time and 
providing them with the benefit of others' expertise. 

Evidence Based Library and Information Practice is an open access journal featuring original 
research articles, evidence summaries, and commentaries. The journal is available entirely 
online and is hosted by the University of Alberta using the Open Journal Systems platform, 
part of the Public Knowledge Project . It reaches an international audience and as of 2009, has 
over 2000 registered readers. The journal launched in January 2006, with issues published 
quarterly. Evidence Based Library and Information Practice is the only current library and 
information studies journal that publishes evidence summaries. 


Evidence summaries 






The evidence summaries published in Evidence Based Library and Information Practice are 
modelled on synopses found in health sciences journals such as ACP Journal Club, Evidence 
Based Nursing and POEMs (patient-oriented evidence that matters) in Essential Evidence 
Plus. In these publications, clinicians can read structured abstracts of clinical research with 
commentaries, and sometimes a bottom line : a statement that answers the question: What does 
this change (or not change) in my practice? 

Each evidence summary in the journal consists of a structured abstract summarizing the 
original research article as well as a commentary of 300 to 400 words appraising the quality of 
the research. Approximately eight evidence summaries are published in every issue, 
summarizing research in all areas of librarianship. The previously published research that 
undergoes appraisal is culled from reputable peer reviewed journals, mainly in library and 
information studies, and occasionally in related fields. The Associate Editor for Evidence 
Summaries regularly reviews approximately ninety journals that publish peer reviewed 
research articles, and chooses research articles that appear relevant for library and information 
studies practitioners. The topics and research settings are chosen in order to appeal to a wide 
ranging readership. 

Evidence summaries are written by a team of writers selected by the Associate Editor on a 
rotational basis, each normally writing four evidence summaries within a two-year period. 

Team members apply to be on the team, and are selected by the Editorial Team based on their 
areas of expertise and their experience with evidence-based practice and critical appraisal skills. 
Evidence summary writers must follow established guidelines (Appendix A) detailing what 
should be included in an evidence summary, and are encouraged to use existing critical 
appraisal tools to aid them in their appraisal. Each evidence summary submitted by a member 
of the team goes through the journal's regular double-blind peer review process with at least 
two peer reviewers providing feedback before the submission is considered for acceptance by 
the Associate Editor. 

The structured abstract of the evidence summary allows practitioners to obtain the essential 
elements of the study without having to read the original, while the commentary portion offers 
an evaluation of its quality and usefulness. The commentary provides the evidence summary 
writer with the chance to comment on the original study's method, the reporting, the research's 
significance within librarianship and, more importantly, how the research may impact 
professional practice. 

Although several publications and electronic resources in the health sciences publish critical 
appraisals of original research, we were unable to find any published reports of research 
analysing their content. Several studies have investigated the use of evidence summaries for 
impact on knowledge and practice (e.g., Grad et al. 2008 1 but these focused on practitioners' 

(in this case, health professionals') perceptions. 

Purpose and objectives 

The purpose of this research study was to identify criticisms, both positive and negative, of the 
library and information studies research literature. The study's objectives were twofold: 

1. To determine the strengths and weaknesses identified by appraisers of research relevant 
to library and information studies, as reported in the commentary section of published 
evidence summaries, and; 

2. To identify commonalities within evidence summary commentaries published to date. 


Methods 



We undertook a directed qualitative content analysis of the commentary portion of evidence 
summaries published in Evidence Based Library and Information Practice from 2006-2008. 
'Qualitative content analysis goes beyond merely counting words or extracting objective 
content from text to examine meanings, themes, and patterns that may be manifest or latent 
in a particular text' fZhang and Wildemuth 2000 : 309). In this way, we were able to identify 
the nature of the criticisms described in the evidence summaries and the tone (positive, 
negative, neutral) of these statements. The emphasis was not on frequencies, but on the 
importance placed on certain characteristics, and on uncovering new aspects of research 
considered important by the evidence summary writers. 

All twelve issues of the journal were included, for a total of 101 evidence summaries. A data 
extraction form was created based upon evidence summary guidelines for writers along with 
elements relating to validity, reliability and applicability from critical appraisal worksheets such 
as CriSTAL fBooth and Brice 2003) . RELIANT ('Koufogiannakis et al. 2006! and the critical 
appraisal tool created by Glynn (2006). The data extraction form fAppendix Bl was designed to 
facilitate both inductive and deductive analysis. The form went through two rounds of pre¬ 
testing, whereby all three researchers independently coded and compared results. Researchers 
discussed areas of confusion or omission, refining the extraction tool until we felt confident 
that the form was clear and straightforward. 

Each commentary was read through and scrutinized independently by two of the researchers 
after which any discrepancies were resolved by the third researcher. Additionally, we looked for 
possible emerging categories, that is, new concepts that arose from our reading of the evidence 
summaries during the analysis but that were not already accounted for in our data extraction 
form. We kept track of these new categories as well. 

The data extraction form focused on areas of validity, reliability and applicability. While these 
terms might seem to reflect a quantitative or positivist approach, the criteria were also 
applicable for research employing interpretive, qualitative inquiry. 

The data extraction form examined five areas that indicate validity. Validity is 'the extent to 
which the results of the research are likely to be free from bias' ( Reynolds 2000 : 25). In the 
commentary portion of each evidence summary, we looked to see if the evidence summary 
writer noted: 

1. the presence of a focused issue or research question; 

2. the possibility of any conflict of interest; 

3. the suitability and replicability of the research method; 

4. the suitability of the population and the sampling method; 

5. the data collection instrument’s validity. 

For each of these, we coded for whether the writer noted these elements as strengths of the 
original study, weaknesses, or if the element was described neutrally. While it was possible for 
the same element to be coded both as a strength and a weakness: this was rare. 

Seven areas touching on reliability were also examined. Reliability is 'the likelihood that this 
study reports something that is reproducible, as opposed to being a "fluke" or chance result' 
fBooth and Brice 2004 : 105). We looked to see if the evidence summary writer noted: 

1. the clarity of results; 

2. the response rate; 

3. the utility and clarity of the analysis; 









4. the compatibility of the analysis to the research design; 

5. whether or not the study results addressed the original research question; 

6. the limitations of the study; and 

7. whether the conclusions were warranted (i.e., based on actual results of the study). 

Finally, we coded for three aspects relating to applicability, 'the extent to which the results are 
likely to impact on practice' fBooth and Brice 2004 : 105). We looked to see if the evidence 
summary writer noted: 

1. the implications for practice reported in the original study; 

2. the applicability to populations beyond those investigated in the original study; and 

3. any local information required to implement the research results. 

Findings 

General attributes 

To put the evidence summaries into context we noted some general attributes of the 101 
summaries we examined, by grouping them by aspects such as domain, setting, source, and 
length. 

In evidence-based library and information studies, domains are used to assist with identifying 
the most applicable resources to search for evidence and to focus a given question. Based on 
the research of Koufogiannakis, Slater and Crumley 12004) . there are six domains into which 
research literature can generally be grouped. 

The evidence summaries were generally representative of the breadth of domains represented, 
with most articles appraised falling in the categories of information access and retrieval or 
collections (see Table 1). Domains are not mutually exclusive, so an article may belong to more 
than one domain. 


Information access and 
retrieval 

26 

Collections 

25 

Education 

21 

Management 

16 

Reference 

14 

Professional issues 

12 


Table 1: Evidence 
summaries by domain 


The setting for the research was also noted, and is represented in Table 2. The highest number 
of research articles that were critically appraised in an evidence summary came from academic 
settings. Again, the setting within which research takes place is not necessarily mutually 
exclusive, as research could be in both a health and academic setting, for example. 


Academic 

45 

Health 

27 

Public 

12 

School 

8 

Any and/or All 

7 

Scholarly publishing 

6 























Special 


2 

Table 2: Evidence 
summaries by setting 

Commentaries within evidence summaries are supposed to be between 300 and 400 words. We 
tabulated word counts for the 101 commentaries included in this study and found that the mean 
length of commentaries was 517 words. These range between 140 and 1220 words, and the 
majority of commentaries (n=39) were between 400 and 499 words. 

Validity 

For the validity aspect of the evidence summaries, the analysis revealed some trends. Elements 
of validity that were most commonly referred to by the evidence summary writers were: 'the 
suitability of the population and the sampling method' (n=8o) and 'the suitability and 
replicability of the research method' (11=76). In both cases, more than two-thirds of the time, 
these were identified by the evidence summary writers as weaknesses of the original research. 
An example of a comment regarding a weak choice of method: 'one wonders if there are other 
variables in the studies that may have also had an impact on the study results'. Similarly, for 
the suitability of the population, another evidence summary writer commented, 'Participants 
were randomly contacted but it is unclear how randomization was done or whether there was 
a self-selection bias in the type of respondent who agreed to participate (response rates were 
not provided)'. 

The only category relating to validity that was commonly viewed as a strength was 'the presence 
of a focused issue or research question'. This was referred to a total of fifty-one times, most 
commonly (n=24) as a strength of the original study. An example of a comment in the evidence 
summary that was coded as a strength is, 'The aims of the study were clear'. This variable was 
noted as a weakness eight times, and for the remainder, comments were neutral. 

The other two aspects of validity were less often mentioned by the evidence summary writers. 
'the data collection instrument's validity' was mentioned in thirty-eight of the evidence 
summaries and as a weakness on nineteen occasions, while 'the possibility of any conflict of 
interest' was mentioned in merely ten (and as a weakness, for seven of these). 

Reliability 

We coded for the seven areas relating to reliability. For all seven areas, these elements were 
noted by the evidence summary writers as weaknesses of the original research. In general, 
reliability was not an area very heavily noted by the evidence summary writers. The area most 
commonly noted (n=57) was 'limitations of the study', with the majority of these limitations 
(11=33) being looked upon negatively. Only in three did evidence summary writers remark on 
this issue as strength, and twenty-three were neutral in their appraisal. 'The utility and clarity 
of the analysis' was noted in forty-one evidence summaries, mostly as a weakness (n=26), and 
'the compatibility of the analysis to the research design' was also seen as a weakness in the 
research being appraised (n=32 out of 37 mentions). We present an illustrative comment from 
one evidence summary discussing the utility of analysis method: 

The study reports several statistically significant results in relation to the 
research questions, yet the analyses seem misinterpreted. For example, self- 
efficacy and use of electronic information jointly contributed 9% of the variance 
of academic performance. A large amount of variance and thus other 
contributing factors (91%) remain unaccounted. Both the R2 and the adjusted 
R2 (0.05531) indicate that these data do not represent a good statistical model. 

The other areas related to reliability were remarked upon much less by evidence summary 


writers, but when they were noted, were generally presented as weaknesses of the original 
study. 

Applicability 

In contrast to aspects of validity and reliability, applicability was an area many evidence 
summary writers noted as a strength of the original research articles. Most commonly noted in 
the commentary section were ' the implications for practice reported in the original study' 
(n=7o) with forty-five studies noted as having this as a strength. Some reported the 
applicability as weakness of the study (n=i4) or neutrally (n=io). The other two areas, 
'applicability to populations beyond those investigated in the original study' and 'any local 
information required to implement the research results' were mentioned less frequently, but 
positively. An example of a positive comment by one evidence summary writer was: 'suggests 
that Web-based tutorials are at least as effective as face to face teaching sessions and that 
these may be successfully delivered either in the classroom or via the Web'. Occasionally, the 
writer noted applicability of the study as a weakness, for example, 'it is not immediately clear 
how this information could be utilized by library practitioners'. 

Other findings of note 

As part of the content analysis, we coded for other elements beyond validity, reliability and 
applicability. First, we verified if the evidence summary writer situated the research article they 
were appraising within a wider context. Forty-eight of the 101 summaries did situate the 
research they were appraising within a wider context, either commenting on research in this 
area in general, noting societal or political aspects affecting the topic, or directly citing relevant 
research (n=3o). In addition, fifty-seven out of the 101 evidence summaries made statements 
noting the overall significance of the original research to library and information studies. 

One category that emerged from the content analysis was the quality of the literature review. 
Eleven commentaries included a statement about the literature review's contribution to the 
research, with seven evidence summaries noting the literature as a strength, three as a 
weakness, and one neutrally. For example, one evidence summary writer noted, 

What is perhaps more valuable in this paper is the extensive use of the research 
literature to inform the various ideas throughout. The literature reviewis robust, 
and the author includes results from previous studies all though the paper to 
strengthen his statements and conclusions. 

Another emerging category dealt with ethical issues. In one evidence summary the lack of 
participant consent was noted, and in another the original article was noted as a duplicate 
publication. These two examples point to weaknesses in research that readers should be aware 
of and that impact the value of research results. One evidence summary writer consistently 
cited the critical appraisal tool that was employed in writing the evidence summaries. 

Finally, for each evidence summary, the length of the commentary section and corresponding 
number of categories were calculated to determine if there might be a relationship between the 
length of the commentary and the number of elements noted. No relationship was apparent, 
that is, evidence summary commentaries that were longer did not contain more elements of 
appraisal than those that were brief. The average number of categories discussed in an evidence 
summary commentary was 7.8 (range between 4 and 14 elements noted). 

Discussion of findings 

This research has several limitations. Firstly, while we decided to include all evidence 
summaries from the journal Evidence Based Library and Information Practice, there were 101 


summaries, written by twenty-eight different writers, some writing up to six or seven within the 
three year period. This may have led to findings that may not be as applicable to those evidence 
summaries published since 2008, as the team has changed over time. 

The evidence summaries included appraisal of articles that were selected mainly by the 
Associate Editor at the time, and were not randomly selected from the library and information 
studies literature. Findings from this research are therefore not representative of strengths and 
weaknesses of the library and information studies literature as a whole, but reflect the emphasis 
placed on the selected research publications by the evidence summary writing team. As other 
scholarly publications begin to produce evidence summaries, and more writers create them, 
more variability among summaries may appear. The evidence summary writers from this 
journal may exhibit similar biases and preferences that may have skewed our findings. In 
addition, despite the structured element of evidence summaries, the writing style of authors 
varied greatly, thus making the content analysis difficult in some cases. For example, one writer 
might be clearly pointing out a study's weakness, while another might be more indirect. This 
results in difficulties in the analysis. This might explain why, in several cases, we had to judge a 
term as neutral. Perhaps the writer thought their statement was obviously negative, but if it was 
not overt, it was coded as neutral. 

One of the researchers (LK) was also an evidence summary writer, and contributed six 
publications to the set of 101. Potential bias was avoided by assigning the other two researchers 
to analyse these documents, and allowed them to resolve any disagreements. 

We believe this analysis will provide insights for library and information studies professionals 
regarding aspects of validity, reliability and applicability that should be considered when 
conducting or reading research. The analysis highlights several areas where overt weaknesses 
were noted by the evidence summary writers. These areas of weakness are ones that those 
conducting research should be aware of and proactively address when conducting their research 
and reporting results, hence strengthening the body of research within our field. While no 
research is perfect, being aware of weaknesses and potential pitfalls should help library and 
information studies professionals address these elements in their own research endeavours. 
Likewise, noted areas of strength provide examples of good practice that new researchers can 
model. 

This study also raised many new questions related to critical appraisal and library and 
information studies research. Aspects of validity and reliability in studies that were critically 
appraised in Evidence Based Library and Information Practice were more often noted as 
weaknesses of the study. Whether this was due to general poor study design or the focus of the 
writer in trying to point out faults rather than positives, is unknown. A follow-up to this 
research could be to interview those who write evidence summaries to determine their goals 
and motivations, and the degree to which they note negative and positive aspects of previously 
published research. As well, we were left to wonder what we could conclude from areas that 
were not noted very frequently by the evidence summary writers. Are these areas not of concern 
because they were dealt with appropriately by the original author, not important enough to 
mention, or simply overlooked? 

Finally, despite the evidence summary writers’ overall negative view towards elements relating 
to validity and reliability, applicability was still widely viewed as positive: Why? It seems logical 
that if validity and reliability are weak in a study, that study’s applicability would therefore be 
weak as well. Is there a lack of connection between the parts of a paper? Were the evidence 
summary writers focused on weaknesses that put findings into question (validity and reliability) 
and not as concerned with aspects of applicability, or were they trying to end their appraisal on 
a positive note as generally applicability is discussed last in the commentary? Alternatively, the 



research articles may have yielded elements that were relevant to practice, even if the actual 
results came from weak methodology. These questions could be looked at more closely by 
subsequent research that combined interviews of the evidence summary writers with closer 
inspection of the text within a paper, using a sample of the total summaries. 

We also suggest that further study investigating library and information studies practitioners' 
perceptions of evidence summaries and their impact on knowledge and practice would be an 
important contribution to this area of research. Doing similar work to what was done in 
medicine by Grad et al . with their information impact scale would relay a sense of the 
importance of the evidence summaries to practitioners who wish to integrate research into their 
learning and decision making. 

Conclusion 

This study shows that evidence summaries published in Evidence Based Library and 
Information Practice between 2006 and 2008 reveal more weaknesses than strengths in the 
library and information studies research that was critically appraised. In general, evidence 
summary writers tend to remark on weaknesses in reports of research relating to validity and 
reliability, yet paradoxically point out strengths with respect to their applicability to practice. 
The exact reasons for this require further qualitative research, to reach a more complete 
understanding of the reasons why the results were as noted. 

These results have implications for the future of evidence summaries published in Evidence 
Based Library and Information Practice. The Editorial Team will need to consider revisions to 
evidence summary guidelines, possibly adding more structure to the commentary section, and 
adding elements such as comment on the literature review and what critical appraisal tool was 
used, a better balance between strengths and weaknesses, and stricter adherence to word 
length. The Editorial Team has already implemented a checklist of items to consider for peer 
reviewers containing more detailed directions on what they should be looking for and 
commenting on in respect to their review of the evidence summary. They have also 
implemented an application procedure for new evidence summary writers that includes writing 
a sample evidence summary before they joined the team. 
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Appendix A: Guidelines for evidence summary writers 

Evidence Summaries are brief critical appraisal reviews of current research articles. The 
summary follows a standardized format to ensure consistency and ease of use for readers. All 
evidence summaries will undergo peer-review to ensure quality. For each summary, the 
following components must be present: 

1. Descriptive title of the review, that is indicative of what we can learn from this research. 

2. Citation information for the article being reviewed. Use MLA format. 

3. Reviewer’s name and contact information (this needs to be left out when submitting for peer 
review) 

4. Structured Abstract to include the following components: 

• Objective - the objective of the study 

• Design - type of research study design used 

• Setting - environment in which the research took place; i.e., large public library in 
Canada, corporate information centre in the United States, small community college in 
Australia. 

• Subjects - the number and characteristics of the subjects that participated in the study; 
i.e.,: 75 senior citizens who were homebound. 

• Methods - a brief paragraph on the research methodology; i.e., students were randomized 
into two groups, with one receiving computer assisted instruction, the other receiving 
traditional lecture/demonstration. At the end of the term students were asked to complete 
a skills test.etc. 

• Main results - state the main outcome of the research study; i.e., e-books were favored by 
2-1 over print books by young adults participating in the focus group. 

• Conclusion - state the conclusion and practice implications for this research study, i.e., 
based on the research results, signage in the library was improved and replaced. A follow¬ 
up study will be conducted to further examine impact of the change. 

5. Commentary - provide 300-400 words of commentary on the quality of the research article. 
What was good about it; what are areas of concern; what could have been improved? 
Commentary should address the strength of the evidence presented, methodology used, issues 
of reliability, validity and applicability. What is the significance of the article to library and 
information practice; what are the practice implications for librarians in the target 
environment? Try to place this research article in the wider context of research available on 
this topic. How can this research be applied to everyday practice? Reviewers should refer to a 
critical appraisal checklist to ensure all important elements of assessment have been taken into 
consideration. 

Appendix B: Data extraction form 

For elements of commentary, indicate, if the item is present in the commentary, if it is 
mentioned as a STRENGTH, or WEAKNESS of the original study, or if it is mentioned 
neutrally. If it is not present, leave the item blank. 

Highlight the corresponding segment(s) in the commentary, and use the code to identify these 
(e.g., V3, ao) 


* Required 


ID * (enter unique ID for Evidence Summary Commentary from document)_ 

Name of coder_ 

Title of Evidence Summary (cut and paste full title of Evidence Summary here) 


Research situated in a wider context the ES writer mentions other research specifically, or 
generally 

Yes, the ES writer situates the study in a wider context 

AND the ES writer names other researchers or other studies specifically 

VALIDITY (vi) focused issue/research question The ES writer discusses the issue/research 
question"s focus 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

VALIDITY (v2) conflict data collection and service providers The ES writer addresses potential 
conflict of interest: those involved in the collection of data also delivering a service to the user 
group. 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

VALIDITY (v3) appropriate and replicable methods The ES writer addresses the research 
methods (appropriateness of methods and level of description) 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

VALIDITY (v4) appropriate population, representative sample The ES writer discusses the 
appropriateness of the population and sampling technique, and/or the level of description of 
the sampling/population 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

VALIDITY (V5) tested or piloted methods/instruments The ES writer discusses the data 
collection instruments' validation, appropriateness, or usage 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

VALIDITY (vo) other validity, but does not fit well into existing categories (please explain / 
provide emerging category) 


RELIABILITY (ri) results clearly explained The ES writer discusses the clarity of the results 



description/explanation of the original author 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (r2) response rate The ES writer discusses the original study’s response rate (as a 
% or proportion) 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

The ES writer includes the number of respondents, but not as a % or proportion 

RELIABILITY (r3) analysis useful and easy to understand The ES writer discusss the usefulness 
of the analysis, and the presentation/organization (e.g., in table, with numbers rather than just 
%s) 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (r4) analysis appropriate and useful (stats or other) The ES writer discusses the 
appropriateness of the analysis method employed 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (rs) results address original research question The ES writer discusses whether 
or not the study's results address the original research question 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (r6) limitations The ES writer discusses the limitations of the study (even if 
restating limitations mentioned by the original author) 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (r7) conclusions based on results and supported by data The ES writer discusses 
to what extent the conclusion are based on the actual results of the study 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

RELIABILITY (ro) other reliability, but does fit well into existing categories (please explain / 
provide emerging category) 


APPLICABILITY (at) implications for practice reported in original study The ES writer 
discusses the original author's statement(s) about implications for practice 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 



APPLICABILITY (a2) to other populations The ES writer discusses the applicability of this 
resaerch to other populations beyond what was discussed by the original author 
+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

APPLICABILITY (a3) additional research needed The ES writer addresses additional local 
information required to move forward 

+ Appraised positively by ES writer (STRENGTH of original study) 

- Appraised negatively by ES writer (WEAKNESS of original study) 

~ Noted by ES writer (NEUTRAL or UNCLEAR appraisal) 

APPLICABILITY (ao) other applicability, but does fit well into existing categories (please 
explain / provide emerging category) 


Overall siginificance of the original study The ES writer addresses the theoretical/overall 
significance of the research (not just practical implications), e.g., using words like "important," 
"contribution to the field," 

The ES writer includes a statement about the overall significance of the research study 
The ES writer includes future research are OTHER emerging category in addition to context, 
validity, reliability, applicability, significance 


Quotable segment(s)? indicate if there is a representative example of one or more of the above 

categories in the commentary 

(copy & pastas or questions to consider 

e here) 

Remarks by the coder 
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